Current error in tests is only in python api that is missing some Java files in its distribution.

ERROR: setUpClass (test_matrix_rand.TestRand)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/baunsgaard/github/systemds/src/main/python/tests/test_matrix_rand.py", line 50, in setUpClass
    cls.sds = SystemDSContext()
  File "/home/baunsgaard/github/systemds/src/main/python/systemds/context/systemds_context.py", line 97, in __init__
    raise Exception("Exception in startup of GatewayServer: " + stderr)
Exception: Exception in startup of GatewayServer: Exception in thread "main" java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper

The reason why it did not work was because the packaged jar file did not contain all the packages required now (due to me updating some dependencies). Therefore i decided to change the python to not decompress the distribution zip but use the environment variable SYSTEMDS_ROOT to locate the target folder and in turn all the jar files. This means we don't have to include java files in our python distribution, and the python execution comes closer to our bin/systemds distribution script.

The down side is that we have to install SystemDS or compile from scratch to make the Python package work, but i think it is fair to do that.

Does this mean the python package (pip install) no longer contains the jar files? If you are using the target (as I already did at some point before) and use all those jars for the python package it becomes huge (>100MB), which I think is something we would like to not do and keep it relatively small.

Does this mean the python package (pip install) no longer contains the jar files? If you are using the target (as I already did at some point before) and use all those jars for the python package it becomes huge (>100MB), which I think is something we would like to not do and keep it relatively small.

Yes it will no longer contain any jar files, making it much smaller.

Shouldn't pip packages not be dependant on other ressources? That would at least make it much simpler to install and I would be much in favor of a single pip install command.

Shouldn't pip packages not be dependant on other ressources? That would at least make it much simpler to install and I would be much in favor of a single pip install command.

I agree, but we have to have the user setup java anyway and in that context i think installing and setting up systemds is fine.
One solution is that on first execution download the dependencies if SYSTEMDS_ROOT not set? I could live with that.

That also sounds like a better option (in my opinion), but then we would have to define a default path because setting an environment variable for the system will be hard. Also we would not like to download all the jars, which would save us no download work, but instead anyway set the jars to download, so why not do it when building the package and distribute it combined? That would also lead to all the data be in a single directory where pip packages are stored. Also setting up java is a single command (we should actually force java 8 execution, which we don't do atm jus java ... right?).

That also sounds like a better option (in my opinion), but then we would have to define a default path because setting an environment variable for the system will be hard.

I think this is just a decision we have to make at some point. there are many good options for install location /etc/systemds/... or /lib/systemds ? If we do make such a decision we could provide a install for Ubuntu, sudo apt install systemds!

Also we would not like to download all the jars, which would save us no download work, but instead anyway set the jars to download, so why not do it when building the package and distribute it combined? That would also lead to all the data be in a single directory where pip packages are stored.

Well, currently i either had to figure out which packages specifically the Python lib require to execute, or use the packages from systemds that was downloaded anyway. To keep it simple in development i chose the later option. I agree that one install command through pip would be better, but i also would like that we don't install, copy or download the jars twice.

Also setting up java is a single command (we should actually force java 8 execution, which we don't do atm jus java ... right?).

We need to move forward towards java 11. but not for this release: Hadoop 3.x and Spark 3.x use/support Java 11.
https://hadoop.apache.org/docs/current/
https://spark.apache.org/docs/3.0.0/

I didn't go through all the comments in the cited PRs, but I'm curious to see how this Spark/Hadoop upgrade impacts the performance. It might not improve anything, but also should not deteriorate. I am not sure if we have a performance suit. Usually, it is better to fire a few performance tests to be very sure for these kinds of upgrades.
Please ignore if that is already done.

I didn't go through all the comments in the cited PRs, but I'm curious to see how this Spark/Hadoop upgrade impacts the performance. It might not improve anything, but also should not deteriorate. I am not sure if we have a performance suit. Usually, it is better to fire a few performance tests to be very sure for these kinds of upgrades.
Please ignore if that is already done.

If you have some scripts you could test it, unfortunately i don't have such a test suite yet, and currently i don't have access to a cluster with these versions install, once we get this we should as you say test it. I would not expect huge performance increases, but the main point of upgrading is not performance but to address the open CVEs and use the versions other ppl are using.

I didn't go through all the comments in the cited PRs, but I'm curious to see how this Spark/Hadoop upgrade impacts the performance. It might not improve anything, but also should not deteriorate. I am not sure if we have a performance suit. Usually, it is better to fire a few performance tests to be very sure for these kinds of upgrades.
Please ignore if that is already done.

If you have some scripts you could test it, unfortunately i don't have such a test suite yet, and currently i don't have access to a cluster with these versions install, once we get this we should as you say test it. I would not expect huge performance increases, but the main point of upgrading is not performance but to address the open CVEs and use the versions other ppl are using.

True, and as you rightly said, we should move forward towards java 11 and Spark 3.x. Maybe not much needed for this PR, but extensive performance tests will be of the highest priority for java version upgrades. I have scripts, but those will not be sufficiently helpful for this, and also the cluster is a problem. I will work on this.
LGTM for this PR.

That also sounds like a better option (in my opinion), but then we would have to define a default path because setting an environment variable for the system will be hard.

I think this is just a decision we have to make at some point. there are many good options for install location /etc/systemds/... or /lib/systemds ? If we do make such a decision we could provide a install for Ubuntu, sudo apt install systemds!

Let's not forget about windows, but I agree that good options exist for all distros.

Also we would not like to download all the jars, which would save us no download work, but instead anyway set the jars to download, so why not do it when building the package and distribute it combined? That would also lead to all the data be in a single directory where pip packages are stored.

Well, currently i either had to figure out which packages specifically the Python lib require to execute, or use the packages from systemds that was downloaded anyway. To keep it simple in development i chose the later option. I agree that one install command through pip would be better, but i also would like that we don't install, copy or download the jars twice.

That was also the option I choose at first, but it seems a bit extreme that even if I choose python for development, as I expect many of the end users will do, I have to download way more jars which I will never make use of. Downloading twice is also a problem, since if I choose to write code in dml and python (how big do we expect the end user overlap to be?) that would be unnecessary. I think we should either make a blacklist of packages to not download/copy or let mvn create use a collection of exactly those packages that we need, in either case it is necessary to figure out which package are needed.

Also setting up java is a single command (we should actually force java 8 execution, which we don't do atm jus java ... right?).

We need to move forward towards java 11. but not for this release: Hadoop 3.x and Spark 3.x use/support Java 11.
https://hadoop.apache.org/docs/current/
https://spark.apache.org/docs/3.0.0/

Then let's keep it as it is for now

Closing, because of plans to update Spark version after 2.0 release of systemds.